Some known messaging systems provide ‘persistent’ messaging, in which message state information and messages are saved to logs and message queue data files in persistent storage (i.e. non-volatile storage such as disk storage). Persistently stored messages are able to survive most failures and restarts of the messaging system. In response to a failure other than a disk failure, the message state information and messages can be recovered from the logged data and persistently-stored queues. The recoverability of persistent messages and state information is a significant factor in achieving assured once-only message delivery.
For example, a messaging manager may save a persistent message to disk storage before confirming to an application program that an operation has been successfully performed on the message. In a typical messaging network, a message-sending application program issues a ‘put_message’ instruction via an API call, to place an outgoing message in a queue. A local messaging manager program manages this queue and communicates with other local application programs, or other messaging managers within a distributed messaging network, to transfer the message to a target recipient. In a persistent messaging system, the local messaging manager saves the message to persistent storage before confirming to the sender application that the ‘put_message’ operation completed successfully. A persistent message or message-related data may be written to disk for every ‘put_message’ and ‘get_message’ that is performed in order to transfer the message to its destination (in some implementations, where the message is outside the current syncpoint-manager controlled ‘unit of work’), or the message may only be logged when a current unit of work is committed. With distributed units of work, it may be possible to avoid writing logs at a number of intermediate points during a message transfer, but a message that is identified as ‘persistent’ will be written to persistent storage at a pre-defined point.
In contrast, ‘non-persistent’ messages are not saved to disk and so they are discarded when a messaging manager experiences a failure and has to restart, as well as when the messaging manager is stopped by an operator command. Although persistent messaging provides great advantages in terms of recoverability and assured message delivery, persistent messaging also has a performance cost. Writing to disk for every ‘put_message’ and ‘get_message’ operation will reduce message throughput and increase application response times, so there is a business justification for handling some messages non-persistently.
Because of the tradeoff between assured message delivery and performance, the WebSphere MQ family of products from IBM Corporation include support for setting persistence or non-persistence as optional attributes of a message and considerable work has been done to improve performance of persistent messaging (WebSphere and IBM are trademarks of International Business Machines Corporation). Performance has been enhanced by use of high performance disk storage, commit processing techniques that optimize performance in the absence of failures, and efficient recovery techniques to deal with failures. For example, U.S. Pat. No. 5,452,430, filed on 23 Mar. 1994 and assigned to IBM Corporation, describes an approach to handling persistent and non-persistent messages that aims to reduce delays during recovery from a system failure, by storing persistent and non-persistent data in separate sets of pages within a single message queue. C. Mohan and D. Dievendorff, “Recent Work on Distributed Commit Protocols, and Recoverable Messaging and Queuing” IEEE Data Engineering Bulletin, 17(1), pages 22-28, 1994, describes developments in the areas of distributed commit protocols and recoverable messaging and queuing.
However, typical messaging systems still have a relatively inflexible specification of persistence. One proposal is to give messaging system users an opportunity to specify a different persistence policy for different defined system states. In such a messaging system, users would benefit from increased granularity of control over persistence, but the persistence behaviour is still fixed when the persistence policy is specified.
When message persistence has been specified as a desired characteristic, typical messaging systems still invest too much of their resources on ensuring message integrity. In many applications, a high proportion of individual messages have too low a business value to justify this investment, but typical prior art solutions are not able to differentiate between messages of differing value or to take account of the costs of persisting or the risks of not persisting.
In addition to messaging systems, the trade-off between performance and persistence also applies to databases, file systems and other persistent systems. In database systems, data is typically saved persistently but it is also known to store some tables without forcing writes to persistent storage, so that the benefits of query/structuring capability can be obtained in a low-overhead storage solution. This move towards ‘lightweight’ database solutions is likely to lead to a greater need for flexible approaches to persistence management. File systems are increasingly used for reliable data—for example holding configuration data for an application server in XML files within a file system. There is increasing demand for transactional file systems, but the current solutions do not provide flexible persistence control.